Overview

This document performs the DIF Analyses used in the section: Identification of Uniform DIF in Select PROMIS Items: Findings from Lordif Analysis.

Load & Setup

Data for analyses

load("data/data_complete.rda")


DIF USA - Germany - Argentina

data_usa_ger_arg <- data_complete

## Select only PFM items
items_usa_ger_arg <- data_usa_ger_arg %>% 
  dplyr::select(starts_with("PF")) %>% 
  dplyr::select_if(~ !any(is.na(.))) # remove items with missing values

##Create language vector
language_usa_ger_arg <- dplyr::recode(data_usa_ger_arg$country, 
                                      usa = "1", 
                                      ger = "2",
                                      arg = "3")


Results

items <- as.data.frame(items_usa_ger_arg)
group <- language_usa_ger_arg
dif_result_nr2 <- lordif::lordif(items, 
                                 group, 
                                 criterion = "R2",
                                 pseudo.R2 = "Nagelkerke",
                                 R2.change = 0.021, # we chose this threshold to match the items identified from the multiverse
                                 minCell = 5)
EM cycles terminated after 500 iterations.
EM cycles terminated after 500 iterations.
summary(dif_result_nr2)
Call:
lordif::lordif(resp.data = items, group = group, criterion = "R2", 
    pseudo.R2 = "Nagelkerke", R2.change = 0.021, minCell = 5)

$criterion
[1] "R2"

$alpha
[1] 0.01

$pseudo.R2
[1] "Nagelkerke"

$R2.change
[1] 0.021

$beta.change
[1] 0.1

$maxIter
[1] 10

$minCell
[1] 5

$stats
   item ncat  chi12  chi13  chi23 beta12 pseudo12.McFadden pseudo13.McFadden
1     1    5 0.0000 0.0000 0.4918 0.0204            0.0046            0.0048
2     2    5 0.0510 0.1467 0.6534 0.0059            0.0006            0.0007
3     3    5 0.0000 0.0000 0.4376 0.0084            0.0038            0.0039
4     4    5 0.0000 0.0000 0.0249 0.0100            0.0059            0.0065
5     5    5 0.0000 0.0000 0.8470 0.0033            0.0027            0.0027
6     6    5 0.0000 0.0000 0.0009 0.0008            0.0043            0.0057
7     7    5 0.0000 0.0000 0.3087 0.0067            0.0047            0.0050
8     8    5 0.0001 0.0000 0.0096 0.0165            0.0017            0.0025
9     9    5 0.0000 0.0000 0.0635 0.0537            0.0109            0.0114
10   10    5 0.0000 0.0000 0.0000 0.0421            0.0090            0.0112
11   11    5 0.0000 0.0000 0.0003 0.1218            0.0371            0.0390
12   12    5 0.0000 0.0000 0.0738 0.0174            0.0025            0.0030
13   13    5 0.0000 0.0000 0.0275 0.0126            0.0057            0.0065
14   14    5 0.0000 0.0000 0.0001 0.0109            0.0068            0.0086
15   15    5 0.0000 0.0000 0.1576 0.0062            0.0111            0.0114
16   16    5 0.0000 0.0000 0.5247 0.0132            0.0085            0.0086
17   17    5 0.0000 0.0000 0.5206 0.0129            0.0026            0.0027
18   18    5 0.0000 0.0000 0.0291 0.0269            0.0125            0.0132
19   19    5 0.0000 0.0000 0.0073 0.0005            0.0045            0.0054
20   20    5 0.0000 0.0000 0.0432 0.0195            0.0021            0.0027
21   21    5 0.0000 0.0000 0.0000 0.0020            0.0039            0.0074
22   22    5 0.0001 0.0000 0.0092 0.0091            0.0017            0.0026
23   23    5 0.0000 0.0000 0.7745 0.1314            0.0413            0.0413
24   24    5 0.0000 0.0002 0.4865 0.0157            0.0021            0.0023
25   25    5 0.0000 0.0001 0.5778 0.0192            0.0027            0.0029
26   26    5 0.0000 0.0000 0.1521 0.0132            0.0063            0.0067
27   27    5 0.0604 0.0208 0.0508 0.0011            0.0007            0.0015
28   28    5 0.0000 0.0000 0.2396 0.0365            0.0075            0.0077
29   29    5 0.0000 0.0000 0.3032 0.0630            0.0102            0.0105
30   30    5 0.0000 0.0000 0.0027 0.0053            0.0104            0.0116
31   31    5 0.0000 0.0000 0.2165 0.0386            0.0092            0.0095
32   32    5 0.0000 0.0000 0.6523 0.1785            0.0509            0.0510
33   33    5 0.0000 0.0000 0.0791 0.0145            0.0028            0.0033
34   34    5 0.0000 0.0000 0.0026 0.0311            0.0132            0.0142
35   35    5 0.0000 0.0000 0.0003 0.0015            0.0051            0.0066
   pseudo23.McFadden pseudo12.Nagelkerke pseudo13.Nagelkerke
1             0.0001              0.0051              0.0053
2             0.0001              0.0007              0.0008
3             0.0002              0.0046              0.0048
4             0.0006              0.0070              0.0077
5             0.0000              0.0031              0.0031
6             0.0014              0.0056              0.0074
7             0.0002              0.0049              0.0051
8             0.0008              0.0029              0.0043
9             0.0005              0.0129              0.0135
10            0.0022              0.0157              0.0195
11            0.0018              0.0507              0.0531
12            0.0005              0.0030              0.0035
13            0.0007              0.0069              0.0078
14            0.0018              0.0083              0.0104
15            0.0003              0.0130              0.0133
16            0.0001              0.0098              0.0099
17            0.0001              0.0034              0.0036
18            0.0007              0.0145              0.0153
19            0.0009              0.0058              0.0069
20            0.0006              0.0025              0.0033
21            0.0034              0.0047              0.0087
22            0.0008              0.0024              0.0035
23            0.0000              0.0636              0.0636
24            0.0001              0.0032              0.0035
25            0.0001              0.0040              0.0041
26            0.0003              0.0130              0.0137
27            0.0008              0.0009              0.0018
28            0.0003              0.0085              0.0088
29            0.0003              0.0143              0.0147
30            0.0012              0.0156              0.0173
31            0.0003              0.0108              0.0111
32            0.0001              0.0678              0.0679
33            0.0004              0.0031              0.0036
34            0.0010              0.0230              0.0248
35            0.0014              0.0079              0.0100
   pseudo23.Nagelkerke pseudo12.CoxSnell pseudo13.CoxSnell pseudo23.CoxSnell
1               0.0001            0.0048            0.0050            0.0001
2               0.0001            0.0007            0.0008            0.0001
3               0.0002            0.0043            0.0045            0.0002
4               0.0008            0.0067            0.0074            0.0007
5               0.0000            0.0029            0.0030            0.0000
6               0.0018            0.0053            0.0069            0.0017
7               0.0002            0.0046            0.0048            0.0002
8               0.0014            0.0028            0.0041            0.0013
9               0.0006            0.0123            0.0128            0.0006
10              0.0038            0.0150            0.0186            0.0036
11              0.0024            0.0465            0.0487            0.0022
12              0.0005            0.0029            0.0034            0.0005
13              0.0009            0.0065            0.0073            0.0008
14              0.0021            0.0078            0.0098            0.0020
15              0.0004            0.0124            0.0128            0.0004
16              0.0002            0.0090            0.0091            0.0002
17              0.0002            0.0031            0.0033            0.0002
18              0.0008            0.0137            0.0144            0.0007
19              0.0011            0.0055            0.0066            0.0011
20              0.0008            0.0023            0.0031            0.0007
21              0.0040            0.0044            0.0082            0.0038
22              0.0012            0.0023            0.0034            0.0011
23              0.0001            0.0608            0.0609            0.0001
24              0.0002            0.0030            0.0032            0.0002
25              0.0002            0.0036            0.0038            0.0002
26              0.0007            0.0124            0.0131            0.0007
27              0.0009            0.0008            0.0016            0.0008
28              0.0003            0.0082            0.0084            0.0003
29              0.0004            0.0132            0.0135            0.0003
30              0.0017            0.0147            0.0163            0.0016
31              0.0003            0.0103            0.0106            0.0003
32              0.0001            0.0649            0.0650            0.0001
33              0.0005            0.0030            0.0034            0.0005
34              0.0018            0.0220            0.0237            0.0017
35              0.0022            0.0075            0.0096            0.0021
   df12 df13 df23
1     2    4    2
2     2    4    2
3     2    4    2
4     2    4    2
5     2    4    2
6     2    4    2
7     2    4    2
8     2    4    2
9     2    4    2
10    2    4    2
11    2    4    2
12    2    4    2
13    2    4    2
14    2    4    2
15    2    4    2
16    2    4    2
17    2    4    2
18    2    4    2
19    2    4    2
20    2    4    2
21    2    4    2
22    2    4    2
23    2    4    2
24    2    4    2
25    2    4    2
26    2    4    2
27    2    4    2
28    2    4    2
29    2    4    2
30    2    4    2
31    2    4    2
32    2    4    2
33    2    4    2
34    2    4    2
35    2    4    2

$flag
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
[13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
[25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE  TRUE FALSE

$flag.raw
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE FALSE
[13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
[25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE  TRUE FALSE


Plot Results

Figure 1

Note. This graph shows smoothed histograms of the physical function levels of study participants from the USA (black solid line), Germany (orange dashed line), and Argentina (purple dotted line) as measured by the PROMIS Physical Function scale (T-Scores) based on the baseline model using the same parameters across countries. There is broad overlap in the distributions, though individuals from the USA and Germany demonstrated lower levels of Physical Function than individuals form Argentina. There was a stronger ceiling effect for the USA distribution at T-Score = 75 than for Germany or Argentina, which increased to T = 80 in the corrected model.

Figure 2

Fig. S5: Graphical display of the item PFM16 “Are you able to pass a 20-pound (10 kg) turkey or ham to other people at the table?” which shows uniform DIF with respect to country.

Note. The top-left graph presents the item characteristic curves (ICCs) for the item, contrasting the USA (black), Germany (orange dashed), and Argentina (purple dotted). The top-right graph illustrates the absolute differences in ICCs between the USA and the other two countries, highlighting that the major disparities for the USA with both Germany and Argentina occur at lower levels of physical function (theta). The bottom-left graph depicts the item response functions for each group, derived from demographic-specific item parameter estimates (slope and category threshold values), which are also annotated on the graph. Finally, the bottom-right graph displays the weighted absolute difference in ICCs (referencing the top-right graph) based on the score distribution of the focal group, the USA, showing a negligible impact.


Figure 3

Fig. S6: Graphical display of the item PFM33 “Are you able to walk across a balance beam?” which shows uniform DIF with respect to country.

Note. The upper-left graph illustrates the item characteristic curves (ICCs) for the item, comparing the USA (black), Germany (orange dashed), and Argentina (purple dotted). The upper-right graph portrays the absolute differences in ICCs between the USA and the other two countries, showing that the primary differences are at lower levels with Germany and at medium levels with Argentina in terms of physical function (theta). The lower-left graph presents the item response functions for each group, based on demographic-specific item parameter estimates (slope and category threshold values), which are also indicated on the graph. Finally, the lower-right graph demonstrates the weighted absolute difference in ICCs (from the upper-right graph) according to the score distribution of the focal group, the USA, suggesting a minimal impact.


Figure 4

Fig. S7: Graphical display of the item PFM46 “Are you able to pull a sled or a wagon with two children (total 100 lbs/50 kg) for 100 yards (100 m)?” which shows uniform DIF with respect to country.

Note. The top-left graph depicts the item characteristic curves (ICCs) for the item, showcasing comparisons between the USA (black), Germany (orange dashed), and Argentina (purple dotted). The top-right graph details the absolute differences in ICCs for the USA against Germany and Argentina, highlighting that the primary differences for Argentina compared to the other two nations occur at medium levels of physical function (theta). The bottom-left graph presents the item response functions for each group, derived from demographic-specific item parameter estimates (slope and category threshold values), which are also noted on the graph. Finally, the bottom-right graph displays the weighted absolute difference in ICCs (referenced in the top-right graph) based on the score distribution of the focal group, the USA, demonstrating a negligible impact.


Figure 5

Fig. S8: Graphical display of the item PFM51 “Are you able to swim laps for 30 minutes at a moderate pace?” which shows uniform DIF with respect to country.

Note. The upper-left graph displays the item characteristic curves (ICCs) for the item, contrasting the USA (black) with Germany (orange dashed) and Argentina (purple dotted). The upper-right graph demonstrates the absolute differences in ICCs for the USA compared to Germany and Argentina, revealing that the differences primarily occur at lower levels with Germany and at medium levels with Argentina in terms of physical function (theta). The lower-left graph presents the item response functions for each group, based on demographic-specific item parameter estimates (slope and category threshold values), also detailed on the graph. Finally, the lower-right graph illustrates the weighted absolute difference in ICCs (referenced in the upper-right graph) according to the score distribution of the focal group, the USA showing a minimal impact.


Figure 6

Fig. 5 Impact of DIF items on test characteristic curves

Note. These graphs show test characteristic curves (TCCs) for USA (black), Germany (orange dashed), and Argentina (purple dotted) using demographic–specific item parameter estimates. TCCs show the expected total scores for groups of items at each physical function level (theta). The graph on the left shows these curves for all of the items (both items with and without DIF), while the graph on the right shows these curves for the subset of these items found to have DIF. These curves suggest that at the overall test level there are some differences in the total expected score for individuals from all three countries. At the same level of ability (i.e., the same theta score on the x-axis), individuals from Germany obtained higher expected observed sum scores than while at the same level of ability individuals from Argentina had lower observed sum scores than the USA.


Figure 7

Fig. 6 Individual–level DIF impact.

Note. These graphs show the difference in score between using scores that ignore DIF and those that account for DIF. The graph on the left shows a box plot of these differences. The interquartile range, representing the middle 50% of the differences (bound between the bottom and top of the shaded box), range roughly from +0.02 to +0.03 with a median of approximately +0.01. In the graph on the right the same difference scores are plotted against the initial scores ignoring DIF (“initial theta”), separately for individuals from the USA, Germany, and Argentina. Guidelines are placed at 0.0 (solid line), i.e., no difference, and the mean of the differences (dotted line). The overall positive values for Germany indicate that accounting for DIF led to slightly lower scores while the overall negative values for Argentina indicate that accounting for DIF led to slightly higher scores. The negative values for the USA to the left of this graph indicate that accounting for DIF led to slightly higher scores for those with low physical function, while the positive values to the right indicate slightly lower scores for those with high physical function. The positive values for Germany to the left of this graph indicate that these effects were particularly pronounced in those with low physical function.

Identified Items

dif_identified_items <- colnames(items[dif_result_nr2$flag])
dif_identified_items
[1] "PFM16" "PFM33" "PFM46" "PFM51"